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USER INTERFACE TAGS FOR A DOCUMENT PROCESSING SYSTEM 

Field of the Invention 

The invention relates to a hardcopy-based computer user interface 
scheme for document processing applications and, more particularly, to a tag- 
based user interface scheme adapted to perform services with a hardcopy 
document dependent on user or sen/ice information stored as printed data on a 
tag associated with the document. 

Background of the Invention 

Many people are most comfortable dealing with documents in hardcopy 
format. In general, hardcopy documents are easier to read, handle, and store 
than documents kept in the digital domain. No special expertise or computer 
hardware is needed. 

However, in general, it is far easier to manipulate documents in the digital 
domain. Text can be indexed, searched upon, reformatted, extracted, and 
otherwise changed. Stored documents can be easily duplicated, without loss of 
quality, and transmitted from person to person (for example, via e-mail). And 
significantly, all of this can be "accomplished without using paper. Moreover, 
digital copiers and scanners are becoming far more prevalent in both office and 
home settings. 

On the other hand, handling documents in the digital domain typically 
requires access to a computer system or network. If the user of the computer 
system does not have a baseline level of expertise or competence in using the 
system, then productivity can suffer. This is a serious impediment to the 
implementation of a "paperless office," in which digital documents are the 
prevailing document type. 
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Accordingly, there is a need to be able to effectively manage documents 
in the digital domain, as well as to ease the transition from hardcopy documents 
to digital documents. 

Previous attempts to facilitate handling digital documents have often used 
traditional user-interface paradigms. For example, when a hardcopy document 
is to be scanned and entered into a document repository, commands to that 
effect are first entered into a computer terminal or scanning device, which then 
performs the desired service with the document. A similar sequence of steps is 
performed when the hardcopy is to be scanned and faxed, scanned and e- 
mailed, scanned and recognized (via optical character recognition software), or 
any of numerous other possibilities. Although the entry of commands can be 
facilitated via user-friendly software or self-explanatory commands, these extra 
steps are still tedious and may still require a certain level of expertise. 
Moreover, the sequence of commands entered may be lost once the operation 
has been performed, and there is a potential for error even with experienced 
users. 

Another possibility is to employ a cover sheet that includes a form for 
specifying commands. The cover sheet is filled out as the user desires (either 
by hand-writing commands or by marking check-boxes, for example), and the 
scanner interprets the commands on the cover sheet and processes the 
following document accordingly. This, too, can be tedious and relatively 
inefficient, as it requires a special-purpose cover sheet to be used for each job. 
It can be inconvenient to maintain a supply of the proper cover sheets. 

Various one- and two-dimensional data codes are known and available to 
be used to store digital data on hardcopy documents. For example, various 
types of barcodes (for example, the familiar UPC symbol used as a retail product 
code) are very well known and are robustly decodable. Other examples of linear 
barcodes are known as Code 39, Code 128, Interleaved 2 of 5, and Postnet. 
Two-dimensional codes, such as the PDF417 code and the UPS MaxiCode used 
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by the United Parcel Service to track packages, for example, are becoming more 
and more widespread. 

Self-clocking glyph codes, such as Xerox DataGlyphs, are attractive for 
embedding machine-readable digital information in images of various types, 
including ordinary hardcopy documents. These codes have substantial 
tolerance to image distortion and noise because the digital information they 
encode is embedded in and fully defined by explicit machine-readable marks 
(i.e., "glyphs," a term used herein which is not intended to be limited to Xerox 
DataGlyphs, but rather is intended to cover all machine-readable marks). These 
glyphs not only encode the information that is embedded in the code, but also 
define the sample clock that is employed to extract that information from the 
code, so they are responsible for the "self-clocking" property of the code as well 
as the distortion and noise tolerance. 

Another known advantage of self-clocking glyph codes is that they 
ordinarily have an unobtrusive visual appearance. This is especially so of codes 
composed of glyphs that are written on a two-dimensional spatially periodic 
pattern of centers, such as a regular lattice-like pattern of centers, because the 
spatial periodicity of the glyphs causes the code to have a more-or-less 
uniformly textured appearance. For example, logically ordered single bit digital 
quanta typically are encoded by respective elongated slash-like glyphs which 
are written on a two-dimensional, spatially periodic pattern of centers in 
accordance with a predetermined spatial formatting rule, with the individual 
glyphs being tilted to the left or right of vertical by approximately +45° and -45° 
for encoding logical "O's" and "1's", respectively. The mutual orthogonality of the 
glyph encodings for the two logical states of these single bit digital quanta 
enhances the discriminability of the code sufficiently to enable the embedded 
information to be recovered, even when the code pattern is written on a 
sufficiently fine grain pattern of center to cause the code pattern to have a 
generally uniform grayscale appearance. It, however, is to be noted that it also 
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has been shown that self-clocking glyph codes can be designed to encode multi- 
bit digital quanta in the glyphs. 

Accordingly, it would be desirable to provide a solution that facilitates the 
use of documents in the digital domain and the transition of documents from 
hardcopy to digital formats. Such a solution should be simple, efficient, 
convenient, and require little or no expertise on the part of the user. 



Summary of the Invention 

The present invention uses a scheme of encoded tags, such as adhesive 
stickers or labels, to serve as the primary user interface in a hardcopy document 
processing system. Such a system would permit a user to specify an action or a 
service to be performed and his own identity simply by applying a sticker to the 
document and placing it into a bin to be scanned. 

The stickers are relatively small and unobtrusive, and in an embodiment 
of the invention use Xerox DataGlyphs to encode the user's identity, a desired 
service, and an optional argument for the service. The user maintains a supply 
of stickers corresponding to a particular service (e.g., "scan and send to my 
personal e-mail account"). When the user desires that service to be performed, 
he simply applies one of the appropriate stickers to the document. 

A computer system performing the invention operates by first accepting 
information on the user's identity. This can be obtained, for example, by 
scanning and analyzing one of the user's business cards. This information is 
stored in a database and is given a unique user ID number. The user ID is 
combined with a desired service code, and the resulting data set is encoded into 
the desired printed data format. The system then prints a customized set of 
labels representing the user ID and service. 

When the user then wishes to have the service specified by his stickers 
performed, he simply applies one of the stickers to a document and places it into 
a scanning queue. The document is scanned, the sticker is identified and 
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decoded, re-associated with the user's information retrieved from the database, 
and the desired service is performed. 

As will be recognized, this system can be adapted to perform numerous 
services and actions, including but not limited to simply scanning and storing the 
document in a repository, faxing the document, converting the document into a 
standard electronic format (such as Microsoft Word format), and others. 

Accordingly, the sticker-based user interface of the present invention is 
convenient and simple, as it does not require the user to enter commands on a 
computer system or fill out a cover sheet for every document to be processed. It 
is efficient, in that it is only necessary for the user to keep a supply of small 
stickers. 



Brief Description of the Drawings 

FIGURE 1 is an exemplary glyph sticker in a configuration according to 
the invention; 

FIGURE 2 is an exemplary block diagram illustrating a data structure 
represented by the glyph sticker of FIGURE 1 ; 
FIGURE 3 is a system-level block diagram; 

FIGURE 4 is a high-level functional flow chart of the sticker creation 
function of the invention; 

FIGURE 5 is a high-level functional flow chart of the sticker processing 
function of the invention; 

FIGURE 6 is a flow chart illustrating the sequence of steps performed in 
locating the sticker as set forth in FIGURE 5; 

FIGURE 7 is a flow chart illustrating the sequence of steps performed in 
decoding the sticker as set forth in FIGURE 5; 

FIGURE 8 illustrates a portion of an exemplary glyph lattice with 
associated glyph bitmaps and lattice vectors; 
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FIGURE 9 is a flow chart illustrating the sequence of steps performed by 
the invention in determining the glyph lattice as shown in FIGURE 7; 

FIGURE 10 is a schematic diagram illustrating the search directions 
employed in searching for glyphs as shown in FIGURE 7; 

FIGURE 11 is a flow chart illustrating the sequence of steps performed in 
finding a seed glyph as in FIGURE 7; 

FIGURE 12 is a schematic diagram illustrating the procedure used to find 
all glyphs as in FIGURE 7; 

FIGURE 13 is a flow chart illustrating the sequence of steps performed in 
finding all glyphs near the seed glyph as shown in FIGURE 7; 

FIGURE 14 is a flow chart illustrating the steps performed in decoding a 
glyph pattern according to the invention. 



Detailed Description of the Invention 

The invention is described below, with reference to detailed illustrative 
embodiments. It will be apparent that the invention can be embodied in a wide 
variety of forms, some of which may be quite different from those of the 
disclosed embodiments. Consequently, the specific structural and functional 
details disclosed herein are merely representative and do not limit the scope of 
the invention. 

Figure 1 illustrates an exemplary glyph sticker pattern 110, which serves 
a paper user interface tag according to the invention. It should be noted that this 
pattern 1 10 is not to any particular scale; it is contemplated to be on the order of 
1 inch (2.5 cm) or less in the vertical direction; it has been substantially enlarged 
here for clarity and to facilitate the identification of details. 

Furthermore, while the user interface tag of the invention is illustrated in 
Figure 1 as an adhesive sticker capable of being applied to documents, it should 
be observed that the pattern 110 may also be applied to forms, slip sheets, 
cover sheets, business cards, identification cards, name tags, and other objects 
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capable of being associated with a document and further capable of bearing 
machine-readable information. For example, in an alternative embodiment of the 
invention, lapel pins bearing the pattern 110 may be associated with a document 
by scanning the lapel pin (e.g., with a hand-held laser scanner) at the same time 
the document is being scanned and processed. 

As shown, the glyph sticker pattern 110 includes several important 
attributes. A substantially rectangular border 111 surrounds the remaining 
features, and is in turn surrounded by white space 113. A glyph field 112 
contains a printed representation of digital data used to perform the goals of the 
invention; the structure and contents of the glyph field 112 will be discussed in 
further detail below. For now, it should be noted that the glyph field 112 of 
Figure 1 is shown as using Xerox DataGlyphs. However, it will be observed that 
any recoverable printed representation of digital information, including but not 
limited to bar codes, one- and two-dimensional data patterns, and optically 
recognizable alphanumeric characters can also be used in alternative 
embodiments of the invention to similar effect. 

Other identifying elements may also be present within the glyph sticker 
pattern 110. For example, the Xerox "Digital X" 114 and the registered mark 
"PaperWare®" 116 are present and are used for aesthetic purposes and as 
trademarks. The phrase "The Person" 118 is also present; it is contemplated 
that this field may be replaced with an indication of the user's name or other 
information for visual identification. There is also a border portion 120 that is 
representative of a folded page. This border portion 120 may be present for 
aesthetic purposes only, or may be used by the invention to facilitate 
determination of the correct orientation of the glyph sticker pattern 110. This 
aspect of the invention will be discussed in further detail below. 

Referring now to Figure 2, an exemplary data structure 210 embodied by 
the glyph field 112 is set forth as shown. The data structure 210 includes a 
service code 212, which in one embodiment of the invention is a single byte 
(eight binary bits). Accordingly, the service code 212 can represent up to 256 
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different possible actions, transformations, and services. Exemplary services 
may include, but would not be limited to, "scan to document repository," "scan 
and send via e-mail," "scan and fax," "scan and print copies," and so forth. An 
indicated service may, without limitation, include a plurality of actions (e.g., scan, 
then recognize characters, then e-mail the text), and may also involve 
transformation of the document from hardcopy to electronic form, and possibly 
back to hardcopy form. Moreover, one or more possible actions might involve 
the removal or alteration of the glyph sticker pattern 110 on the digitized image 
of the document, as its presence is no longer necessary (and may detract from 
the document's appearance) once it has been acted upon by the system and 
method of the invention. 

A service argument 214 provides a coded argument to the service code 
212. In an embodiment of the invention, the argument 214 is an additional byte 
(eight bits) of information. For example, certain services may require a numeric 
argument (e.g., "scan and print copies" followed by an argument of "10" will print 
ten copies). Other services may require a differently coded argument (e.g., 
"scan and fax" followed by an argument of "2" may represent a command to fax 
the document to the user's home fax number, as opposed to an office fax 
number or, perhaps, an alternate office fax number, both of which would have 
different argument numbers). Many other uses of the service argument 214 may 
also be envisioned without departing from the scope of the invention. 

An identity code 216 comprises most of the rest of the data structure 210. 
In an embodiment of the invention, the identity code includes sixteen bytes of 
identity information (sufficient to encode a unique identification number for each 
member of nearly any conceivable population) and eight bytes of security 
information, the latter rendering the identity code essentially tamper-proof. The 
identity information is formed from the network address (either an Ethernet 
address or an IP address) of the machine issuing the identity code, a time 
stamp, and a sequence number. Although a number formed in this manner is 
not guaranteed to be unique under all circumstances (for example, if the 
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database of sequence numbers is lost and the same time stamp is coincidentally 
used twice), it is quite robust and sufficiently unique for the purposes of the 
invention. It should also be noted that the identity code may be formed in any of 
numerous other ways, all of which would be acceptable for use with the present 
invention. 

The data structure 210 also includes error-correction and synchronization 
information throughout the specified data fields in a manner well known in the art 
and not shown in Figure 2. 

Figure 3 represents a system-level overview of the invention, broken 
down into several high-level functional elements. A "front end" portion 310 
receives information from a data input 312. The data input 312 may comprise a 
keyboard, a business card scanner, a voice recognition device, or any other 
means of data collection operative to receive data representative of the user's 
identity. The data input 312 may be expected to receive the user's name, 
company, title, address, one or several telephone numbers, one or several e- 
mail addresses, as well as other possible identification information (such as a 
U.S. Social Security number or a conference ID number). This user information 
is then received by an identity processor 314, which stores the information as a 
record in a database hosted by a database server 316. Concurrently, the user 
information is processed by the identity processor 314 to produce the identity 
code 216 (incorporating the identity and security information discussed above). 
The identity code 216 and any desired optional service code 212 and service 
argument 214 are then encoded and printed on a printer 318 or other hardcopy 
output device in communication with the identity processor 314. 

When the user wishes to process a document, he attaches a glyph sticker 
according the invention onto the -document and places the document into a 
scanner 320, which is part of a "back end" 319 of the system. The scanner is 
preferably an automated scanner capable of handling multiple jobs and multi- 
page documents without user intervention. However, any other type of digitizing 
apparatus (such as flatbed scanners, digital copiers, and hand-held 
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scanners) would also be usable in this application. The scanner 320 reads the 
document and formulates a bitmap representative of the document (and the 
glyph sticker on the document). 

An action processor 322 reads the bitmap received from the scanner 320, 
identifies and decodes the glyph sticker, and accesses the database server 316 
to determine the identity of the user. The desired service may be known from 
accessing a list of possible services stored locally on the action processor 322, 
or may also be determined by accessing a service database on the database 
server 31 6, or alternatively may be inferred simply from the identity of the user. 

Based on the user's identity and the desired service, the action processor 
322 then causes the desired action to be performed, which may involve the 
generation of a transformed document by an output device 324. The output 
device 324 is characterized generally here, but as discussed above, may 
comprise a hardcopy printer, a facsimile machine (or modem capable of sending 
fax messages), a network connection for e-mail, a connection to a document 
repository, a digital storage device (such as a floppy disk drive), or an 
aggregation of some or all of these and other functions. 

While the system of Figure 3 is illustrated as being broken down into 
several functional blocks, it should be noted that the functions depicted can be 
performed on a single computer system or by several computer systems 
connected via a communications network. As a specific example, the database 
server 316 may be local to the identity processor 314 and the action processor 
322, or it may be a separate processor in communication with the other elements 
via a communications network, such as the Internet or a corporate intranet. 

As suggested above by Figure 3, the front end processing of the system, 
namely the creation of glyph sticker patterns 110 based on user information and 
a desired service, takes place as a sequence of steps. These steps are shown 
in the flow chart of Figure 4. 

First, the user (or another person) inputs information (step 410) into the 
system, typically via the data input 312 (Figure 3). All of the information is then 
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stored in the database (step 412) on the database server 316. A service is then 
selected (step 414). The service may be chosen from a "menu" of possible 
services, or in one embodiment of the invention, only a single service may be 
operative. In the latter case, the service code 212 and the service argument 214 
(Figure 2) may optionally be omitted from the glyph sticker pattern 110. 

The user identity code 216, the service code 212, and the service 
argument 214 are then encoded and formed into a glyph field 112 representative 
of the information (step 416). The newly-created customized glyph field 112 is 
then printed any number of times as glyph sticker patterns 110 onto any desired 
number of stickers (step 41 8) for eventual use by the user. 

Figure 5 is a flow chart illustrating what happens after the user has 
applied a sticker to a document and presented it for scanning. First, the 
document is scanned (step 510) and the resulting bitmap is retained for 
processing. The original hardcopy document may, at this time, be returned to 
the user, as it is no longer needed by the system of the invention. 

The glyph sticker pattern 1 10 is then located on the document (step 512). 
In a preferred embodiment of the invention, the glyph sticker pattern is located 
via a rectangle-matching method described below in conjunction with Figure 6. 
In an alternative embodiment of the invention, the user is directed to place the 
glyph sticker at a certain location on the document (e.g., the upper right corner 
of the first page), thereby simplifying the operations required to locate the glyph 
sticker pattern 110. In another alternative embodiment, the sticker pattern 110 
can be identified anywhere on a document by segmenting the scanned 
document bitmap into text and graphics portions, and in the line-graphics 
portions, locating the right-angled lines that comprise the border of the glyph 
sticker pattern 110. A method for accomplishing this is set forth in detail in U.S. 
Patent No. 5,202,933 to Bloomberg, entitled "SEGMENTATION OF TEXT AND 
GRAPHICS," which is hereby incorporated by reference as though set forth in 
full. To facilitate this, half-tones and finely textured graphics regions (such as 
lithographed photographs) may also be segmented out (and away from the line 
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graphics) via the methods taught in U.S. Patent Nos. 5,065,437 and 5,131,049. 
It should be noted that the glyph sticker pattern 110 may be present in nearly 
any orientation; accordingly, the segmentation method should be adapted to 
check several different orientations (by using differently-shaped structuring 
elements in the morphological operations used in the incorporated '933 patent 
document to identify horizontal and vertical lines) before it is determined that no 
sticker is present. Finally, a Hausdorff-distance-based algorithm may also be 
used; it should be noted that this method, like the morphological method 
referenced above, is rotationally sensitive and hence relatively computationally 
inefficient. 

Once the glyph sticker pattern 110 has been located, the data within the 
glyph field 112 is then decoded (step 514). This is accomplished via methods 
that will be discussed in further detail below, in connection with Figures 7-14. 
The data is then associated (step 516) with user identification information 
(received from the database server 316) and a desired service, and the service 
is performed (step 518) be means known in the art and discussed above. 

A presently preferred method used to locate user interface tags (e.g., the 
glyph sticker pattern 110 of Figure 1) within a digitized image is illustrated in 
Figure 6. This method can be used to identify and locate objects that are 
approximately rectangular in shape, such as the iconic representation of a page 
with one corner folded over, as used as the glyph sticker pattern 110 in an 
embodiment of the invention (see Figure 1). Such rectangular objects can be 
characterized by a height dimension and a width dimension. The identification 
method can identify the desired pattern in any orientation and with relatively 
small scale variations (such as those occurring in the printing and digitizing 
processes), even when image noise is present. 

The method is operative on monochromatic images (i.e., binary images 
represented with only one bit per pixel). If the digitized image is in some other 
format (such as color or grayscale), it should first, as a precursor to the method 



12 



"Express Mail" No. EMB^44383US Patent Application 

Attorney Docket No. D/98703 

set forth in Figure 6, be converted to a binary format, typically via a threshold 
function or by dithering. 

The characteristics (e.g., resolution) of the digitizing device used are 
expected to be known. Therefore, because the absolute size of the glyph sticker 
pattern 110 (Figure 1) is known, its image size can be calculated. Accordingly, 
the expected width w and the expected height h of the pattern 110, in pixels, is 
either known or can be easily calculated. With width and height tolerances (Aw 
and Ah, respectively) to account for minor variations in scale, as stated above, a 
minimum expected width would be ww, = (w - Aw), a maximum expected width 
would be w^ = (w + Aw), a minimum expected height would be = (h- Ah), and 
a maximum expected height would be /w = (h + Ah). In a presently preferred 
embodiment, Ah = (0.i)h and Aw = (O.l)w; stated another way, the tolerances are 
10% of the respective height and width. It should be noted, however, that other 
tolerances ranges may also be operative, and that different tolerances may be 
used for the width and height dimensions. 

Accordingly, and based on the foregoing, the minimum and maximum 

expected diagonal measurements would be d,^ = yj(w- Aw) 2 + (h- Ah) 2 and 

= 7(w+Aw) 2 +(h + Ah) 2 , respectively. 

After a suitable digitized image is available, all connected components 
within the image are initially identified (step 610). A connected component is a 
set of pixels of a single value (e.g., the value representing black), wherein it is 
possible to form a path from any pixel of the set to any other pixel in the set 
without leaving the set (e.g., by traversing only black pixels). In general terms, a 
connected component may be either "4-connected" or "8-connected." In the 4- 
connected case, the path can move in only horizontal or vertical directions, so 
there are four possible directions. Accordingly, two diagonally adjacent black 
pixels are not 4-connected, unless there is another black pixel horizontally or 
vertically adjacent, serving as a bridge between the two. In the 8-connected 
case, the path between pixels may also proceed diagonally. A presently 
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preferred embodiment of the invention uses 8-connected components, but it is 
contemplated that 4-connected components may also be identified and used. 

Because the border 111 of the glyph sticker pattern 110 (Figure 1) is 
formed from single group of contiguous black pixels, it should be observed that 
the border 111 comprises a single connected component. In a preferred 
embodiment of the invention, the white space 113 around the border 111 is 
sufficiently wide to ensure that the border 1 1 1 is not inadvertently connected 
with some other portion of the hardcopy document to which the glyph sticker 
pattern 110 is applied. Moreover, the border 111 is also sufficiently thick to 
ensure that it is not inadvertently broken into discrete components by image 
noise. 

The connected components within the image are identified by means 
known in the art (i.e., by starling with a single black pixel within the image, 
recursively locating all connected pixels until the connected component is fully 
defined, and repeating until all black pixels within the image belong to a 
connected component). However, other means of identifying connected 
components may also be used to equivalent effect. 

Each connected component is then processed separately. If there are 
any connected components remaining to process (step 611), then the method 
continues as follows. If the connected component is too small (step 612), that is, 
if its width or height is less than a minimum expected value, then the connected 
component is rejected (step 614). In an embodiment of the invention, the 
minimum expected value, used for both height and width, is the smaller of 
and HWn to account for possible rotation of the pattern 110. Likewise, the 
connected component is rejected (step 614) if it is too large (step 616), and its 
width or height exceeds the maximum expected value by a substantial amount. 
In an embodiment of the invention, the maximum expected value, for both height 
and width, is substantially larger than the greater of and vtw to account for 
possible rotation. When the pattern 110 (or any rectangle) is oriented at an 
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angle, its width and height may appear to be larger than expected, approaching 
or equaling d^. Accordingly, a buffer is built into the latter check. 

Eight extreme points, one for each of eight "compass positions," are then 
selected (step 61 8) from the set of pixels comprising the connected component 
C. Each point has a position represented by an (x, y) coordinate pair, and 
represents a pixel of the connected component C that extends furthest in the 
selected direction (north, northeast, east, southeast, south, southwest, west, or 
northwest) within the image plane. Each extreme point is chosen as follows. 



N = (x N ,y N )eC\y N <yV(x,y)eC 
E = (x E ,y E )eC\x E >xV(x,y)<=C 
S=(x s ,y s )eC\y s >yV(x,y)<=C 
W = (x w ,y w )<= C\x w <xV(x,y)e C 
NE = (x m ,y NE )eC\(x NE -y NE )>(x-y)\/(x,y)G C 
SE = {x SE ,y SE )eC\ +> SE )>(i+y)V(xj)eC 
SW = (x sw ,y sw )eC\(x sw -y„) < (x- y) V (x,y)e C 
= (x NW ,y NW )eC\(x NW +y NW )<(x + y)V (x,y)e C 

It should be noted that various optimizations can be performed in 
identifying the eight extreme points. For example, if the connected component C 
is broken down into horizontal runs of contiguous pixels, then only the leftmost 
pixel in each run need be considered as a candidate for the NW, W, and SW 
extreme points, and only the rightmost pixel in each run need be considered as a 
candidate for the NE, E, and SE extreme points. Moreover, if the horizontal runs 
are ordered vertically, then only the endpoints of the uppermost and lowermost 
runs need be considered for the N and S extreme points. It should further be 
noted that, when the border 111 (Figure 1) is oriented precisely at any 45° 
interval from upright, then there may be many pixels within the connected 
component that satisfy one or more of the criteria for extremeness set forth 
above. This is not a problem; any satisfactory pixel may be chosen. For 
example, there is no single south-most pixel in the border 1 1 1 of Figure 1 , so 
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any pixel along the bottom edge may be chosen. Moreover, it is not a problem 
for a single pixel to be used as multiple extreme points. Referring again to 
Figure 1, for example, a bottom-right corner pixel 122 may be used as the 5, SE, 
and E extreme points without any impact on the described method. 

If the connected component C is the border 1 1 1 (namely, a rectangle with 
one missing corner), then three of the extreme points will contain rectangular 
corner points (the folded-over corner will be offset), and the other five will 
contain irrelevant information. However, the method described herein has not 
yet determined whether the connected component C is the border 1 1 1 , or if so, 
which extreme points represent its corners; that is determined as set forth below. 

Consider, then, all eight extreme points. If the distance between any two 
diametrically opposed extreme points satisfies the diagonal length criteria (step 
620), then the connected component is a border candidate. That is, if 
d^^N-S^d^, or if d^^E-W^d^, or if d^^E-SW^d^, or if 

d^ < \\SE-NW\\ < d^ , then a border candidate has been found. Otherwise, the 

connected component C is rejected (step 614). 

Based on the position of the diagonal, the other two potential corners of 
the connected component are then identified (step 622) and considered. If the 
diagonal was found between either the N-S or E-W extremities, then the values of 
N, E, 5, and W are used for p u Pi, p$, and p 4 , representing the four corner points 
of the border candidate, respectively. Similarly, if the diagonal was found 
between either the NE-SW or SE-NW extremities, then the values of NE, SE, SW, 
and NW are used for p u pi, p^ and p 4f the four corner points of the border 
candidate, respectively. 

Then, relationships among the four points p u Pi* Ps, and p* are analyzed to 
determine whether a rectangular shape is present (step 624). In particular, the 
distances between pi and /? 2 , pi and p 3 , p% and p 4 , and p 4 and p x are all 
considered. At least one distance (of the four possible) should approximate the 
expected width, and at least one adjacent (but not the opposing) distance should 
approximate the expected height. 
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That is, if either 

(Kin ^ -Pl\^ W max) 0T ( W mm ^ \\P 3 ~ P*\\ * W max » ^ 
((>U> * ||/>2 - ^ *b« ) 0r C'nrin ^ |/>4 ~ Pi \\ ^ ^na*)) 

or 

^ ||Pl ~ P 2 | ^ 0r ^mi* ^ ||P3 " P 4 | ^ ''max)) ^ 

^ \\p 2 -PiW^w^) or (w^ < \\p 4 -p x \< W^)) 

is true, then the connected component C is potentially the border 111, and it is 
added to a list (step 626) of possible positions. If not, the connected component 
C is, once again, rejected (step 614). 

As stated above, a list of potential positions is generated from all border 
candidates that satisfy the foregoing criteria. The entire list is then passed to 
the decode process (step 514 of Figure 5). Presumably, only one border 
candidate will contain valid glyph (or other) data. If more than one border 
candidate includes valid data, then the first one found is used. 

It should be noted that the foregoing method of identifying and locating 
the substantially rectangular border 111 of the glyph sticker pattern 110, in any 
orientation and with minor variations in scale, would be applicable to and useful 
in numerous other image-processing applications. It is contemplated that the 
method illustrated in Figure 6 and described in detail above may be used to 
identify and locate nearly any substantially rectangular pattern within a digitized 
image, either as a part of or separately from the other features and capabilities 
described herein. 

Furthermore, it will be appreciated that minor variations of this method, 
readily apparent to those skilled in the art, may also be used to identify and 
locate various parallelogram, rhombus, trapezoid, and irregular quadrilateral 
patterns in addition to rectangles and rectangle-like shapes. In these alternative 
embodiments, either one or both of the diagonals can be checked; the two 
diagonals may have different expected lengths. In addition, the method can be 
extended to identify and locate n-sided polygons, by identifying extreme points at 
In evenly-spaced compass positions, and thereafter checking for vertices at 
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alternating extreme points. Where n is an odd number, it should be noted that 
any diagonal found will not be diametric; however any expected distance (or set 
of expected distances) between adjacent or non-adjacent vertices, in nearly any 
n-sided polygon, can be used in a method according to the invention to identify 
the polygon within an image. 

Figure 7 illustrates the sequence of steps performed in recognizing a 
glyph field 112 (Figure 1) and translating it into binary data. First, the glyph 
lattice is determined (step 710). Essentially, glyph lattice vectors (Figure 8) 
along with skew and scale information need to be determined; this operation is 
illustrated in Figure 9 and described in further detail below. Second, a seed 
glyph is found (step 712). The recognition process begins by finding a single 
"good" glyph, the seed glyph; see Fig. 11. Third, all of the glyphs are found 
(step 714) by recursively examining the seed glyph's neighbors (as shown in 
Figs. 12-13). The lattice vectors direct the search and help perform the mapping 
from image space to a matrix of glyph values. As will be shown below, these 
steps can all be performed regardless of the glyph field's rotation. 

Once all of the glyphs have been located, the proper rotation of the glyph 
field 112 is determined (step 716). As can be seen in Figure 1, the preferred 
glyph configuration comprises "slash" patterns at +45° and -45° orientations. 
Accordingly, the glyph field 112 may appear to be valid at orientations of 0°, 90°, 
180°, and 270°. Therefore, the binary patterns represented by the glyph field 
112 are processed to determine the correct rotation. In one embodiment of the 
invention, a synchronization pattern is embedded into the glyph field 112; this 
pattern will only be reconstructed correctly when the field 112 is in the correct 
orientation. For a method of disambiguating the glyph field orientation, see the 
specification and drawings of U.S. Patent No. 5,449,895 to Hecht and Stearns, 
entitled "EXPLICIT SYNCHRONIZATION FOR SELF-CLOCKING GLYPH 
CODES," which is hereby incorporated by reference as though set forth in full 
herein. In particular, see columns 15-16 of the '895 patent for a detailed 
description of rotational disambiguation. It should be noted that the position of 
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the corner 120 (Figure 1) may also be used in an alternative embodiment of the 
invention to determine the correct orientation of the glyph field 1 12. 

As shown in Figure 8, a glyph image 810 can be defined by two lattice 
translation vectors 812 and 814 and two (or more) bitmaps 816 and 818 that 
specify the information that is to be represented at each lattice site. This 
definition is analogous to defining a crystal in solid-state physics. This method 
of specifying the glyph image allows for a wide variety of structures, but it also 
provides a scheme for handling arbitrary rotations, scales, lattices, and bitmaps. 
For instance, rotation of the glyph image 810 simply corresponds to a rotation of 
the translation lattice vectors 812 and 814. Just as translation vectors would 
define the displacement from glyph center to glyph center during construction of 
the glyph image, the translation vectors also define the displacement from glyph 
center to glyph center during the recognition process. Viewing the glyph image 
as a lattice allows the recognition scheme to handle a wide variety of glyph 
formatting schemes while handling rotation and scale variations. 

The method for determining the glyph lattice is based upon building up an 
average picture of what the local neighborhood of the glyph is, and from this 
average, determining the lattice vectors. In a preferred embodiment of the 
invention, this is implemented as follows (and as illustrated in Figure 9). 

In the glyph image (that is, the area within the border 111), identify some 
number of pixels in the image as seed pixels for processing (step 910). These 
seed pixels may be, for example, spaced on a lattice of N x N pixels throughout 
the image, where N is of the order of 10 - 50 (the spacing may be chosen so that 
there are approximately 100 seed pixels throughout the glyph image 81 0). 

Then, starting at each seed pixel, find the local minimum intensity in the 
image nearest that seed (step 912). If no local minimum is found within an 
appropriate distance of the seed pixel (e.g. 10 pixels), then move to the next 
seed pixel. The local minima, i.e. the darkest points in the bitmap, typically 
correspond to glyphs. 
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Once a local minimum is found, it is determined whether there is sufficient 
image contrast (step 914) between the region within 1 to 2 pixels of the minimum 
(i.e., the center), and the region 3 to 4 pixels from the minimum (i.e., the 
surrounding area). If the contrast is too small compared to the average image 
intensity surrounding the local minimum, then the method abandons further 
computation with the present local minimum, and moves to the next seed pixel in 
the glyph image (if there are any). This step, which may be left out if desired, is 
employed to screen out spurious local minima in the image resulting from noise 
and other non-glyph material. 

Otherwise, the grayscale image surrounding the local minimum is added 
(step 916) to a composite image, which initially is filled with pixels of value zero. 
This composite image, which is built up during the processing, thus becomes a 
sum of the bitmap values around a number of local minima in the glyph image. 
The composite image is typically on the order of 20 x 20 pixels in dimension. 

If there are any seed pixels remaining to be processed (step 918), then 
the process is repeated. 

After finding the local minima associated with all of the seed pixels, the 
composite image is analyzed (step 920) to determine the average glyph 
locations. For example, the center of the composite image will necessarily be a 
local minimum, as it is composed of many images whose centers were minima. 
The nearest local minima to the center in the composite image will then 
correspond to the average nearest neighbor positions of the glyphs in the glyph 
lattice. Determination of these minima in the composite image will therefore 
result in knowledge of the configuration of the glyph lattice; the lattice vectors 
812 and 814 can then be derived. 

The method described above and in conjunction with Figure 9 is very 
robust, due to the averaging properties of building up the composite image. As 
in any signal averaging procedure, it is expected that in the composite image, 
pixels corresponding to the locations of glyphs will sum coherently, while image 
noise will not. In practice, it has been found that the average glyph lattice 
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parameters of an image may be obtained robustly using the above procedure, 
even when a large number (>50%) of the glyphs are obscured or eradicated by 
image noise or overwriting. In addition, the above procedure works under any 
amount of image skew. 

From the glyphs' lattice vectors 812 and 814, a search direction list 1010 
is generated. See Figure 10. The search direction list is a list of corresponding 
relative displacements in image space 1012 and data space 1014 for finding 
neighboring glyphs. From a known glyph in image space, the search direction 
list 1010 provides information on where the neighboring glyphs should be 
located. And for each translation in image space (1016, 1018, 1020, and 1022), 
there is a corresponding relative displacement in data space (1024, 1026, 1028, 
and 1030). Accordingly, for a known lattice structure, rotation, and scale, the 
relative movements in image space can be specified without changing the 
corresponding movement in data space. This method of defining the search 
allows the process to accommodate a wide variety of glyph lattices, rotations, 
and scales. It should be noted that the search direction list is also extendable to 
search for second-nearest-neighbor glyphs, or non-rectilinear search patterns. 

The operations performed in finding a seed glyph for the recognition 
process will be discussed in connection with Figure 11. Finding a good seed 
glyph is critical. If the seed glyph is actually a spurious center, the process will 
quickly fail. Preferably, the process looks for a seed glyph that has at least four 
neighboring glyphs. This gives some assurance that the chosen seed is not 
spurious. 

The seed glyph is found by looking at random locations in the images. At 
a chosen random location (step 1 1 10), a set of correlation filters is applied over 
a 3 x 3 neighborhood of pixels adapted to cover the area of a single glyph (step 
1112). The 3x3 neighborhood accommodates noise and small variations in the 
glyph lattice. The correlation filters that are applied depend upon the image 
skew determined from finding the lattice vectors 812 and 814. If the skew is 
between -22.5° and 22.5°, the following correlation filter kernels are applied: 
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If the skew is between -45° and -22.5°, the following kernels are applied: 





"0 


0 


0" 




"0 


i 


0" 




1 


1 


1 




0 


i 


0 




0 


0 


0 




0 


i 


0 



If the skew is between 45° and 22.5°, the following kernels are applied: 
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The two correlation kernels are applied separately over the 3 x 3 neighborhood 
where a glyph is expected. Since the glyphs typically appear black, the 
minimum value of the two correlations is observed, and the difference between 
the two correlations is formed (step 1114) and compared to a threshold (step 
1116): 

v = min(K 0 * 1^) - min(A' 1 * ) 

where * denotes correlation. If the absolute value of v is less than a preset 
threshold (step 1118), the location does not contain a glyph. If v is positive, the 
glyph has a value of 0, and if v is negative, the glyph has a value of 1 . 

Once a glyph is found, the method looks for neighboring glyphs (step 
1120) using the search direction list described above. The method stops looking 
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for a seed glyph once it has found a glyph with four neighboring glyphs (step 
1122). 

The final step in the recognition process is to determine the values of the 
individual glyphs and place these values into a matrix. The invention uses the 
search direction list (Figure 10) to find new glyphs and to properly map their 
values into a matrix. The process, which is shown in Figure 13, begins at the 
seed glyph. The seed glyph is placed into a dynamic data matrix and the 
location of the glyph in image space and data space are placed onto a search 
FIFO (first-in-first-out) list (step 1310). 

The search FIFO list controls the clustering. As new glyphs are found, 
they are placed at the end of the search FIFO list. Essentially, the search FIFO 
list contains the locations in the image that need to be searched for neighboring 
glyphs. When the search FIFO list is empty, the process stops. 

Beginning with the seed glyph, if any glyphs are present in the FIFO list 
(step 1312), the method pulls a glyph location from the FIFO list (step 1314). Its 
value is determined (step 1316) and placed into the data matrix. The value of 
the glyph is determined by using the same correlation kernels used to find the 
seed glyph. The method then searches outward (step 1318) looking for 
neighboring glyphs using the search direction list (see Fig. 12(a)); neighboring 
glyphs are placed into the FIFO list (step 1320). The method generates a 
cluster of known glyphs that grows outward as it searches for new glyphs around 
the perimeter of the cluster (Fig. 12(b)). The outcome of the procedure is a 
data matrix full of data values 1210 (Fig. 12(c)) representative of the glyph 
image 810. 

This method has several desirable features. First, it can handle groups of 
glyphs with arbitrary boundaries. The glyph block need not be rectangular. 
Second, it provides a clean method of dealing with scale and rotation variations. 
Finally, it is tolerant to intensity gradients and markings placed on top of the 
glyphs. It should be noted that although a FIFO list is used in the presently 
preferred embodiment of the invention, other search methods, including those 
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that use a LIFO (last-in-first-out) stack or a list based on some other ordering 
scheme, such as position within the image, can also be used with similar effect. 

Finally, as discussed above, the data matrix is used as shown in Figure 
14. First, the data matrix is rotationally disambiguated as described above (step 
1410). The binary values in the data matrix are re-mapped accordingly (step 
1412), which may involve re-ordering the matrix and inverting the data values, as 
necessary based on the rotation. The service code is extracted from the 
resulting data (step 1414), as is the user code (step 1416), and the invention 
handles the service as discussed above in conjunction with Figure 3. 

While certain exemplary embodiments of the invention have been 
described in detail above, it should be recognized that other forms, alternatives, 
modifications, versions and variations of the invention are equally operative and 
would be apparent to those skilled in the art. The disclosure is not intended to 
limit the invention to any particular embodiment, and is intended to embrace all 
such forms, alternatives, modifications, versions and variations. 
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